A constrained regularization approach for input-driven recurrent neural networks

We introduce a novel regularization approach for a class of inputdriven recurrent neural networks. The regularization of network parameters is constrained to reimplement a previously recorded state trajectory. We derive a closed-form solution for network regularization and show that the method is capable of reimplementing harvested dynamics. We investigate important properties of the method and the regularized networks and show that the regularization improves task-specific generalization on a combined prediction and non-linear sequence transduction task. The approach has strong theoretical and practical implications.